As a baseline for DCLM, we conduct extensive experiments and find that model-based filtering is key to assembling a high-quality training set.

Thirty years ago, a peace-loving Austrian theologian spoke to Peter Thiel about the apocalyptic theories of Nazi jurist Carl Schmitt. They've been a road map for the billionaire ever since. For a full two years now, the billionaire has been on the circuit, spreading his biblically inflected ideas about doomsday through a set of variably and sometimes visibly perplexed interviewers. He has chatted onstage with the economist podcaster Tyler Cowen about the (the scriptural term for "that which withholds" the end times); traded some very awkward on-camera silences with the New York Times columnist Ross Douthat; and is, at this very moment, in the midst of delivering a four-part, off-the-record lecture series about the Antichrist in San Francisco. Depending on who you are, you may find it hilarious, fascinating, insufferable, or horrifying that one of the world's most powerful men is obsessing over a figure from sermons and horror movies. But the ideas and influences behind these talks are key to understanding how Thiel sees his own massive role in the world--in politics, technology, and the fate of the species. And to really grasp Thiel's katechon-and-Antichrist schtick, you need to go back to the first major lecture of his doomsday road show--which took place on an unusually hot day in Paris in 2023. No video cameras recorded the event, and no reporters wrote about it, but I've been able to reconstruct it by talking to people who were there. The venue was a yearly conference of scholars devoted to Thiel's chief intellectual influence, the late French-American theorist René Girard. On the evening of the unpublicized lecture, dozens of Girardian philosophers and theologians from around the world filed into a modest lecture hall at the Catholic University of Paris. And from the dais, Thiel delivered a nearly hourlong account of his thoughts on Armageddon--and all the things he believed were "not enough" to prevent it. By Thiel's telling, the modern world is scared, way too scared, of its own technology. Our "listless" and "zombie" age, he said, is marked by a growing hostility to innovation, plummeting fertility rates, too much yoga, and a culture mired in the "endless Groundhog Day of the worldwide web." But in its neurotic desperation to avoid technological Armageddon--the real threats of nuclear war, environmental catastrophe, runaway AI--modern civilization has become susceptible to something even more dangerous: the Antichrist. According to some Christian traditions, the Antichrist is a figure that will unify humanity under one rule before delivering us to the apocalypse. For Thiel, its evil is pretty much synonymous with any attempt to unite the world. "How might such an Antichrist rise to power?" Thiel asked.

Days after New Hampshire voters received a robocall with an artificially generated voice that resembled President Joe Biden's, the Federal Communications Commission banned the use of AI-generated voices in robocalls. The 2024 United States election would be the first to unfold amid wide public access to AI generators, which let people create images, audio and video – some for nefarious purposes. Institutions rushed to limit AI-enabled misdeeds. Sixteen states enacted legislation around AI's use in elections and campaigns; many of these states required disclaimers in synthetic media published close to an election. The Election Assistance Commission, a federal agency supporting election administrators, published an "AI toolkit" with tips election officials could use to communicate about elections in an age of fabricated information.

How are AI assistants being used in the real world? While model providers in theory have a window into this impact via their users' data, both privacy concerns and practical challenges have made analyzing this data difficult. To address these issues, we present Clio (Claude insights and observations), a privacy-preserving platform that uses AI assistants themselves to analyze and surface aggregated usage patterns across millions of conversations, without the need for human reviewers to read raw conversations. We validate this can be done with a high degree of accuracy and privacy by conducting extensive evaluations. We demonstrate Clio's usefulness in two broad ways. First, we share insights about how models are being used in the real world from one million Claude.ai Free and Pro conversations, ranging from providing advice on hairstyles to providing guidance on Git operations and concepts. We also identify the most common high-level use cases on Claude.ai (coding, writing, and research tasks) as well as patterns that differ across languages (e.g., conversations in Japanese discuss elder care and aging populations at higher-than-typical rates). Second, we use Clio to make our systems safer by identifying coordinated attempts to abuse our systems, monitoring for unknown unknowns during critical periods like launches of new capabilities or major world events, and improving our existing monitoring systems. We also discuss the limitations of our approach, as well as risks and ethical concerns. By enabling analysis of real-world AI usage, Clio provides a scalable platform for empirically grounded AI safety and governance.

We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters. As a baseline for DCLM, we conduct extensive experiments and find that model-based filtering is key to assembling a high-quality training set. The resulting dataset, DCLM-Baseline enables training a 7B parameter language model from scratch to 64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the previous state-of-the-art in open-data language models, DCLM-Baseline represents a 6.6 percentage point improvement on MMLU while being trained with 40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53 natural language understanding tasks while being trained with 6.6x less compute than Llama 3 8B. Our results highlight the importance of dataset design for training language models and offer a starting point for further research on data curation.

Darren Holly steers coils of steel through Pentaflex, a manufacturer of parts for heavy trucks, in Springfield, Ohio. Darren Holly steers coils of steel through Pentaflex, a manufacturer of parts for heavy trucks, in Springfield, Ohio. Americans who endured the brutal 2007-2009 recession and slow recovery now are seeing an economic sunrise: Wages are up, jobs are growing and more families are lifting themselves up out of poverty. And yet, dark clouds are still hanging over millions of Americans. No set of sunny statistics can help an unemployed coal miner in Kentucky pay the mortgage.